Patient identification systems and methods

ABSTRACT

Disclosed techniques relate to identifying subjects in digital images. In some embodiments, intake digital images ( 404 ) are acquired ( 1002 ) that capture a first subject. A subset of the intake digital images is selected ( 1004 ) that depict multiple different views of the first subject&#39;s face. Based on the selected subset of intake digital images, first subject reference templates are generated and stored in a subject database ( 412 ). Later, a second subject is selected ( 1008 ) for identification within an area. Associated second subject reference templates are retrieved ( 1010 ) from the subject reference database. Digital image(s) ( 420 ) that depict the area are acquired ( 1012 ). Portion(s) of the digital image(s) that depict faces of subject(s) in the area are detected ( 1014 ) as detected face image(s). A given detected face image is compared ( 1016 ) to the second subject reference templates to identify the second subject ( 1018 ) in the digital image(s) that capture the area.

TECHNICAL FIELD

The present disclosure is directed generally, but not exclusively, toidentifying people in digital images (including streams of digitalimages). More particularly, but not exclusively, various methods andapparatus disclosed herein relate to identifying people in digitalimages (or streams thereof) so that those people can be located in areassuch as waiting rooms of a hospital.

BACKGROUND

There are a number of scenarios in which it may be desirable toautomatically identify people (or “subjects”) based on digital imagesthat capture scenes containing people. For example, when patients visita hospital, they typically are registered, triaged, and then sent to anarea such as a waiting room to wait for hospital resources such asphysicians to become available to examine and/or treat the patients.Being able to automatically identify individual patients may be helpfulfor continuing to monitor their conditions (e.g., for deterioration)while they wait for allocation of medical resources. It may also behelpful for determining if/when patients left without being seen (LWBS).Automatically identifying people based on digital images may also beuseful in a variety of other contexts, such as airports, train stations,border crossings, gyms and fitness centers, various businesses, etc.

In some contexts, it may be desired to identify individual subjects indigital images that contain multiple subjects. For example, digitalimages captured by a camera in a waiting room are likely to depict, inaddition to waiting patients, other people such as friends, relatives,etc. that might be waiting with the patients. Face detection techniquesmay detect all the faces in the digital images, but it may not be clearwhich faces belong to patients and which belong to others. Moreover,subjects in monitored areas such as waiting rooms are not likely goingto be looking at the camera. Instead they may be looking at theirphones, magazines, each other, etc. Thus, even when depicted faces aredetected, the detected faces as depicted in their raw state may not beideal for identifying subjects. In addition, the light conditions in thearea may vary across time (e.g., daytime versus nighttime) and/or acrossthe physical space.

SUMMARY

The present disclosure is directed to methods, systems, and apparatusfor automatically identifying people depicted in acquired digitalimages. As one non-limiting example, a plurality of triaged patients maywait in a waiting room until they can be seen by an emergency medicinephysician. The patients may be included in a patient monitoring queue(also referred to simply as a “patient queue”) that is ordered orranked, for instance, based on a measure of acuity associated with eachpatient (referred to herein as a “patient acuity measure”) that isdetermined based on information obtained/acquired from the patient by atriage nurse, as well as other data points such as patient waiting time,patient presence, etc. One or more “vital sign acquisition cameras”mounted in the waiting room may be configured to periodically performcontactless and/or unobtrusive acquisition of one more updated vitalsigns and/or physiological parameters from each patient. These updatedvital signs and/or physiological parameters may include but are notlimited to temperature, pulse rate, oxygen saturation (“SpO₂”),respiration rate, posture, perspiration and so forth.

In order to identify a particular patient from which the vital signacquisition camera(s) should acquire updated vital signs, techniquesdescribed herein may be employed to match so-called “subject referencetemplates”—e.g., digital images that depict a variety of different viewsof a subject's face—to a person contained in a scene captured in one ormore digital images acquired by one or more vital sign acquisitioncameras, e.g., from a relatively wide field of view (“FOV”). Moregenerally, techniques described herein may be implemented in variouscontexts to identify subjects depicted in digital images (e.g., singleimages and/or streams of digital images, such as video feeds), e.g., bycollecting subject reference templates associated with each subject tobe monitored and later using those subject reference templates toidentify the subject in subsequently captured digital images.

Generally, in one aspect, a method may include: acquiring a plurality ofintake digital images that capture at least a first subject; selecting,from the plurality of intake digital images, a subset of intake digitalimages that depict multiple different views of a face of the firstsubject; generating, based on the selected subset of intake digitalimages, first subject reference templates, wherein the first subjectreference templates are stored in a subject database in association withinformation related to the first subject, and the subject databasestores subject reference templates related to a plurality of subjects;selecting a second subject to identify within an area; retrieving secondsubject reference templates related to the second subject from thesubject reference database; acquiring one or more digital images thatdepict the area; detecting, as one or more detected face images, one ormore portions of the one or more digital images that depict faces of oneor more subjects in the area; comparing a given detected face image ofthe detected one or more detected face images to the second subjectreference templates; and identifying, based on the comparing, the secondsubject in the one or more digital images that capture the area.

In various embodiments, the area may include a waiting room, the intakeimages may be acquired using a first camera that is configured tocapture a registration or triage area, and the digital images thatdepict the area may be acquired using a second camera that is configuredto capture the waiting room. In various embodiments, the comparing mayinclude applying the given detected face image as input across a trainedmachine learning model to generate output that indicates a measure ofsimilarity between the given detected face image and the second subject,wherein the machine learning model is trained based at least in part onthe second subject reference templates. In various embodiments, thetrained machine learning model may take the form of a lineardiscriminant analysis model. In various embodiments, the method mayfurther include retraining the machine learning model in response to anew subject being added to the subject database or an existing subjectbeing removed from the subject database. In various embodiments, thetrained machine learning model may be trained based on the subjectreference templates related to the plurality of subjects.

In various embodiments, one or more of the subset of intake digitalimages may be selected based on being sufficiently dissimilar to one ormore other intake digital images. In various embodiments, the method mayfurther include normalizing the one or more face images so that eachdetected face image depicts a frontal view of a face. In variousembodiments, the normalizing may include geometric warping.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts discussed in greater detail below (provided suchconcepts are not mutually inconsistent) are contemplated as being partof the subject matter disclosed herein. In particular, all combinationsof claimed subject matter appearing at the end of this disclosure arecontemplated as being part of the subject matter disclosed herein. Itshould also be appreciated that terminology explicitly employed hereinthat also may appear in any disclosure incorporated by reference shouldbe accorded a meaning most consistent with the particular conceptsdisclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to the sameparts throughout the different views. Also, the drawings are notnecessarily to scale, emphasis instead generally being placed uponillustrating the principles of the disclosure.

FIG. 1 schematically illustrates a general process flow for monitoringpatients identified in digital images using disclosed techniques, inaccordance with various embodiments.

FIG. 2 illustrates an example environment in which various components ofthe present disclosure may implement selected aspects of the presentdisclosure, in accordance with various implementations.

FIG. 3 depicts an example scenario in which disclosed techniques may bepracticed, in accordance with various embodiments.

FIG. 4 depicts example components and operations for performing variousaspects of the present disclosure.

FIG. 5 depicts an example of how subject reference templates may beselected from intake digital images for generation of subject referencetemplates, in accordance with various embodiments.

FIG. 6 depicts an example of how a subject may be detected enteringand/or leaving a camera's field of view, in accordance with variousembodiments.

FIG. 7 depicts one example of how a detected face image may benormalized, e.g., to be front-facing, in accordance with variousembodiments.

FIG. 8 depicts, in greater detail than FIG. 5, an example of how subjectreference templates may be selected from intake digital images, inaccordance with various embodiments.

FIG. 9 depicts one example of pose-adaptive face image matching, inaccordance with various embodiments.

FIG. 10 depicts an example method for performing selected aspects of thepresent disclosure, in accordance with various embodiments.

FIG. 11 depicts components of an example computer system.

DETAILED DESCRIPTION

FIG. 1 schematically illustrates generally how patients may be monitoredusing disclosed techniques. In particular, operations and actions aredepicted that may occur in a pre-waiting room area, such as at apre-waiting room area(s) 102, which may include reception and/orregistration, and/or a triage station or booth. In addition, operationsand actions are depicted that may occur in a waiting room 104. It shouldbe understood that the sequence of FIG. 1 is not meant to be limiting,and other sequences are possible.

At block 106, a new patient may enter and/or approach pre-waiting roomarea(s) 102, e.g., after checking in at a reception desk (not depicted).At block 108, the new patient may be registered. Registration mayinclude, for instance, collecting information about the patient such asthe patient's name, age, gender, insurance information, and reason forvisit. Typically, but not exclusively, this information may be manuallyinput into a computer by medical personnel such as receptionist orregistrar. In some embodiments, one or more reference digital images ofthe patient may be acquired, e.g., by a camera that is integral with acomputing device operated by the triage nurse, by a standalone camera,and/or by a vital sign acquisition camera (in which case at least somevital signs may be optionally acquired at registration). As will bedescribed in more detail below, in some embodiments, the digital imagesacquired by the camera during registration at block 108 may be referredto as “intake digital images.” Subsets of these intake digitalimages—and in some cases, selected sub-portions of these images thatdepict, for instance, faces—may be selectively retained as “subjectreference templates” that can be used later to identify patients (ormore generally, “subjects”) in areas such as waiting room 104.

In many instances, the triage nurse additionally may acquire variousinitial vital signs and/or physiological parameters at block 110 usingvarious medical instruments. These initial vital signs and/orphysiological parameters may include but are not limited to bloodpressure, pulse, glucose level, SpO₂, photoplethysmogram (“PPG”),respiration rate (e.g., breathing rate), temperature, skin color, and soforth. While not depicted in FIG. 1, in some embodiments, otherinformation may be gathered at triage as well, such asacquiring/updating a patient's medical history, determining patientallergies, determining patient's use of medications, and so forth. Insome embodiments, the patient may be assigned a so-called “patientacuity measure,” which may be a measure that is used to rank a severityof the patient's ailment, and in some instances may indicate ananticipated need for emergency room resources. Any number of commonlyused indicators and/or clinician decision support (“CDS”) algorithms maybe used to determine and/or assign a patient acuity measure, includingbut not limited to the Emergency Severity Index (“ESI”), the TaiwanTriage System (“TTS”), the Canadian Triage and Acuity Scale (“CTAS”),and so forth. For example, in some embodiments, vital signs of thepatient may be compared with predefined vital sign thresholds stored ina system database, or with published or known vital sign values typicalfor a given patient age, gender, weight, etc., to determine thepatient's initial patient acuity measure and/or the patient's initialposition in the patient queue. In some embodiments, variousphysiological and other information about the patient may be applied asinput across a trained model (e.g., regression model, neural network,deep learning network, etc.), case-based reasoning algorithm, or otherclinical reasoning algorithm to derive one or more acuity measures. Insome embodiments, the information used for deriving the acuity measuremay include or even be wholly limited to vitals or other informationthat may be captured by the vital sign acquisition camera. In someembodiments, the information used for deriving the acuity measure mayalternatively or additionally include information such as informationfrom a previous electronic medical record (“EMR”) of the patient,information acquired from the patient at triage, information fromwearable devices or other sensors carried by the patient, informationabout other patients or people in the waiting room (e.g., vitals ofothers in the room), information about family members or othersassociated with the patient (e.g., family member EMRs), etc.

Once the patient is registered and/or triaged, at block 112, the patientmay be sent to waiting room 104. In many scenarios, the operations ofFIG. 1 may occur in slightly different orders. For example, in someinstances, a patient may first be registered, then go to a waiting roomuntil they can be triaged, and then be send to a doctor some time aftertriage (either immediately or after being sent back to the waitingroom). In some situations, such as emergency situations (e.g., duringdisasters), patients may go straight to triage and then to a doctor, andmay only be registered later when the patient has been stabilized.

At block 114, it may be determined, e.g., using one or more cameras,sensors, or input from medical personnel, that a patient has left thewaiting room. Block 114 may include scanning each person currentlywithin the waiting room (e.g., as part of a seeking function thatattempts to locate the patient once the patient is at the top of a queueof patients for which vitals are to be captured, such as an execution ofblock 120 described below, or cycling through each person in the room tocapture vitals, as multiple executions of the loop including blocks 118and 120 described below) and determining that the patient was notlocated. In some embodiments, the system may wait until a predeterminednumber of instances of the patient missing is reached or a predeterminedamount of time has passed during which the patient is missing before thepatient is deemed to have left the waiting room to account for temporaryabsences (e.g., visiting the restroom or speaking with clinical staff).For example, the patient may have been taken into the ER proper becauseit is their turn to see a doctor. Or the patient's condition may haveimproved while they waited, causing them to leave the hospital. Or thepatient may have become impatient and left to seek care elsewhere.Whatever the reason, once it is determined that the patient has left thewaiting room for at least a threshold amount of time, at block 116, thepatient may be deemed to have left without being seen and may bereleased from the system, e.g., by removing them from a queue in whichregistered patients are entered.

At block 118, a patient in waiting room 104 may be selected formonitoring. For example, in some embodiments, a database (e.g., subjectreference database 412 in FIG. 4) storing registration informationobtained at blocks 108-110 may be searched to select a patient havingthe highest patient acuity measure or a patient having the highestacuity measured that has not been monitored recently, as may bedetermined by a time threshold set for all patients or set (e.g.,inversely correlated) based on the acuity measure. In other embodiments,registration information associated with a plurality of patients in thewaiting room may be ranked in a patient monitoring queue, e.g., by theirrespective patient acuity measures, in addition to or instead of othermeasures such as waiting times, patient presence in the waiting room(e.g., missing patients may be selected for monitoring more frequentlyto determine whether they should be released if repeatedly absent), etc.In yet other embodiments, patient acuity measures may not be consideredwhen ranking the patient monitoring queue, and instead onlyconsiderations of patient waiting times, patient presence, etc., may beconsidered.

However such a patient monitoring queue is ranked, in some embodiments,the first patient in the queue may be selected as the one to bemonitored next. It is not required (though it is possible) that thepatient monitoring queue be stored in sequence of physical memorylocations ordered by patient acuity measures. Rather, in someembodiments, a ranked patient monitoring queue may merely include a rankor priority level value associated with each patient. In other words, a“patient monitoring queue” as described herein may refer to a “logical”queue that is logically ranked based on patient acuity measures, waitingtime etc., not necessarily a contiguous sequence of memory locations.Patients may be selected for monitoring at block 118 in an order oftheir respective ranking in the patient monitoring queue.

At block 120, the patient selected at block 118 may be located inwaiting room 104. In various embodiments, one or more cameras, such asone or more vital sign acquisition cameras (not depicted in FIG. 1, seeFIGS. 2, and 3), that are deployed in or near waiting room 104 may beoperated (e.g., panned, tilted, zoomed, etc.) to acquire one or moredigital images of patients in waiting room 104. As will be described inmore detail below, those acquired digital images may be compared to oneor more reference patient images (often referred to herein as “subjectreference templates”) captured during registration at block 108.

At block 122, one or more vital sign acquisition cameras mounted orotherwise deployed in or near waiting room 104 may be operated toperform unobtrusive (e.g., contactless) acquisition of one or moreupdated vital signs and/or physiological parameters from the patientselected at block 118 and located at block 120. These vital signacquisition cameras may be configured to acquire (without physicallycontacting the patient) a variety of different vital signs and/orphysiological parameters from the patient, including but not limited toblood pressure, pulse (or heart) rate, skin color, respiratory rate,SpO₂, temperature, posture, sweat levels, and so forth. In someembodiments, vital sign acquisition cameras may be equipped to performso-called “contactless methods” to acquire vital signs and/or extractphysiological information from a patient may be used as medical imagedevices. Non-limiting examples of such cameras are described in UnitedStates Patent Application Publication Nos. 20140192177A1, 20140139656A1,20140148663A1, 20140253709A1, 20140235976A1, and U.S. Pat. No.9,125,606B2, which are incorporated herein by reference for allpurposes.

At block 124, it may be determined, e.g., by one or more componentsdepicted in FIG. 2 (described below), based on a comparison of theupdated vital sign(s) and/or physiological parameters acquired at block122 to previously-acquired vital signs and/or physiological parameters(e.g., the initial vital signs acquired at block 110 or a previousiteration of updated vital signs/physiological parameters acquired bythe vital sign acquisition cameras), whether the patient's condition haschanged. For example, it may be determined whether the patient's pulserate, respiratory rate, blood pressure, SpO2, PPG, temperature, etc. hasincreased or decreased while the patient has waited. If the answer isno, then control may proceed back to block 118, and a new patient (e.g.,the patient with the next highest patient acuity measure) may beselected and control may proceed back to block 120. However, if theanswer at block 124 is yes (i.e. the patient's condition has changed),then control may pass to block 126. In some embodiments, the patient'scondition may be represented (at least partially) by the same acuitymeasure used for purposes of determining monitoring order.

At block 126, it may be determined (again, by one or more components ofFIG. 2) whether a medical alert is warranted based on the changedetected at block 124. For example, it may be determined whether achange in one or more vital signs or patient acuity measures satisfiesone or more thresholds (e.g., has blood pressure increased above a levelthat is considered safe for this particular patient?). If the answer isyes, then control may pass to block 128. At block 128, an alarm may beoutput, e.g., to a duty nurse or other medical personnel, that thepatient is deteriorating. The medical personnel may then check on thepatient to determine if remedial action, such as immediately admittingto the ED to see a doctor, is warranted. In some embodiments, controlmay then pass back to block 118. However, if the answer at block 126 isno, then in some embodiments, control may pass back to block 118.

FIG. 2 depicts example components that may be used to practice disclosedtechniques, in accordance with various embodiments. A hospitalinformation system 240 may be of the type that is commonly found inhospitals, doctor's offices, and so forth. Hospital information system240 may be implemented using one or more computing systems that may ormay not be connected via one or more computer networks (not depicted).Hospital information system 240 may include, among other things, aregistration module 242, a triage module 244, a release module 246, andan alarm module 248. One or more of modules 242-248, or any other moduleor engine described herein, may be implemented using any combination ofhardware and software, including one or more microprocessors executinginstructions stored in memory. For example, the registration module 242may include registration instructions implementing the functionalitydescribed herein in connection with registration executing on aprocessor while the triage module 244 may include triage instructionsimplementing the functionality described herein in connection withtriage executing on the same processor. Similar underlying hardware andsoftware may be used to implement other “modules” described herein.

Registration module 242 may be configured to receive, e.g., as manualinput from a duty nurse, registration information of new patients. Thismay include, for instance, the patient's name, age, insuranceinformation, and so forth. Triage module 244 may be configured toreceive, e.g., as manual input from a duty nurse or directly fromnetworked medical equipment, vital signs such as those described aboveand/or other physiological data, such as weight, height, the patient'sreason for the visit, etc. In various embodiments, vital signs receivedby triage module 244 and/or a patient acuity measure (e.g., ESI in FIG.2) may be associated with corresponding patient information received byregistration module 242, e.g., in one or more databases (not depicted)associated with hospital information system 240.

Alarm module 248 may be configured to receive information indicative ofvarious events, such as patient deterioration, and raise various alarmsand/or alerts in response. These alarms and/or alerts may be outputusing a variety of modalities, including but not limited to visualoutput (e.g., on display screens visible to hospital personnel),intercom announcements, text messages, emails, audio alerts, hapticalerts, pages, pop-up windows, flashing lights, and so forth. Modules242-248 of hospital information system 240 may be operably coupled,e.g., via one or computer networks (not depicted), to a hospitalinformation system interface 250 (“H.I.S. Interface” in FIG. 2).

Hospital information system interface 250 may serve as an interfacebetween the traditional hospital information system 240 and a patientmonitoring system 252 configured with selected aspects of the presentdisclosure. In various embodiments, the hospital information systeminterface 250 may publish, e.g., to other modules of the patientmonitoring system 252, various information about patients such asregistration information, patient acuity measures (e.g., ESI),prescribed and/or administered medications, whether a patient has beenreleased, various alarms/alerts, and so forth. As will be describedbelow, in some embodiments, these publications may be provided to anevent publish and subscribe (“EPS”) module 270, which may thenselectively store them in database 272 and/or selectively publish themto other modules of patient monitoring system 252. In some embodiments,hospital information system interface 250 may additionally oralternatively subscribe to one or more alerts or publications providedby other modules. For example, hospital information system interface 250may subscribe to alerts from deterioration detection module 268, e.g.,so that hospital information system interface 250 may notify appropriatecomponents of hospital information system 240, such as alarm module 248,that a patient is deteriorating. EPS is just one of many possibleprotocols that could be used for communication among system components,and is not meant to be limiting.

Patient monitoring system 252 may include a variety of components thatfacilitate monitoring of patients in an area such as waiting room 104 toensure that patients are served in a manner conducive with their actualmedical condition. Patent monitoring system 252 may include, forinstance, a patient capture module 254 that interfaces with one or morecameras 256, a patient queue module 258, a patient identification module260, a dynamic calibration module 262, a face/torso acquisition module264, a vital signs measurement module 266, a deterioration detectionmodule 268, the aforementioned EPS module 270, and one or more databases272, 274. As noted above, each of modules 250, 254, and 258-270 may beimplemented using any combination of hardware and software. And whilethese modules are depicted separately, that is not meant to be limitingor to suggest each is implemented on a separate piece of hardware. Forexample, one or more modules may be combined and/or omitted, and one ormore modules may be implemented on one or more computing systemsoperably connected via one or more computer networks (not depicted). Thelines depicted connecting various components of FIG. 2 may representcommunication channels accessible to these components. Thesecommunication channels may be implemented using any number of networkingor other computer communication technologies, such as one or more buses,Ethernet, Wi-Fi, Bluetooth, Z-Wave, ZigBee, cellular communication, andso forth.

Patient monitoring system 252 may also include one or more vital signacquisition cameras 276 that are configured to acquire, from somedistance from a patient, one or more vital signs and/or physiologicalparameters of the patient. Examples of such vital sign acquisitioncameras were described above. In various embodiments, a vital signacquisition camera 276 may be a pan-tilt-zoom (“PTZ”) camera that isoperable to pan, tilt, and zoom so that different parts of an area suchas waiting room 104 are contained within its FOV. In this manner, it ispossible to scan the area being monitored to locate different patients,so that updated vital signs and/or physiological parameters may beacquired unobtrusively.

Patient capture module 254 may receive, from one or more cameras 256,one or more signals carrying captured image data of a patient. Forexample, in some embodiments, patient capture module 254 may receive avideo stream from camera 256. Patient capture module 254 may performimage processing (e.g., face detection, segmentation, shape detection todetect human form, etc.) on the video stream to detect when a patient ispresent, and may capture one or more reference digital images of thepatient (e.g., the intake digital images described below) in response tothe detection. In some embodiments, the reference digital images may becaptured at a higher resolution than individual frames of the videostream, although this is not required. In some embodiments, camera 256may be a standalone camera, such as a webcam, a PTZ camera (e.g., 276),and so forth, that is deployed in or near pre-waiting room area(s) 102.Subsets of the intake digital images captured by camera 256 may be usedto generate subject reference templates that are associated withpatients (and more generally, “subjects”) and used later to identifypatients in the area being monitored.

Patient queue module 258 may be configured to establish and/or maintaina priority queue, e.g., in a database, of the order in which patients inthe area should be monitored. In various embodiments, the queue may beordered by various parameters. In some embodiments, patients in thequeue may be ranked in order of patient acuity measures (i.e. bypriority). For example, the most critical patients may be placed at thefront of the queue more frequently than less critical patients. In someembodiments, updated vital signs may be acquired from patients waitingin the area being monitored, such as waiting room 104, in an order ofthe queue. In other embodiments, updated vital signs may be acquiredfrom patients in a FIFO or round robin order. In other embodiments,updated vital signs may be acquired from patients in an order thatcorresponds to a predetermined scan trajectory programmed into vitalsign acquisition camera 276 (e.g., scan each row of chairs in order).

Patient identification module 260 may be configured with selectedaspects of the present disclosure to use one or more digital imagescaptured by vital sign acquisition camera 276 (or another camera that isnot configured to acquire vital signs unobtrusively), in conjunctionwith subject reference templates captured by patient capture module 254,to locate one or more patients in the area being monitored (e.g.,waiting room 104). Patient identification module 260 may analyzeacquired digital images using various techniques described below toidentify and locate patients (subjects). FIGS. 4-10, described below,demonstrate various aspects of various techniques that may be employedas part of recognizing/identifying/locating patients, or more generally,subjects, in any context.

In some embodiments, patient identification module 260 may search anarea being monitored for particular patients from which to obtainupdated vital signs. For example, patient identification module 260 maysearch the area being monitored for a patient selected by patient queuemodule 258, which may be, for instance, the patient in the queue havingthe highest patient acuity measure. In some embodiments, patientidentification module 260 may cause vital sign acquisition camera(s) 276to scan the area being monitored (e.g., waiting room 104) until theselected patient is identified.

Dynamic calibration module 262 may be configured to track the use ofvital sign acquisition camera(s) 276 and calibrate them as needed. Forinstance, dynamic calibration module 262 may ensure that whenever vitalsign acquisition camera 276 is instructed to point to a particular PTZlocation, it always points to the exact same place. PTZ cameras may bein constant or at least frequent motion. Accordingly, their mechanicalcomponents may be subject to wear and tear. Small mechanicalerrors/biases may accumulate and cause vital sign acquisition camera 276to respond, over time, differently to a given PTZ command. Dynamiccalibration module 262 may correct this, for instance, by occasionallyrunning a calibration routine in which landmarks (e.g., indicia such assmall stickers on the wall) may be used to train a correction mechanismthat will make vital sign acquisition camera 276 respond appropriately

Once a patient identified by patient queue module 258 isrecognized/located by patient identification module 260, face/torsoacquisition module 264 may be configured to pan, tilt, and/or zoom oneor more vital sign acquisition cameras 276 so that their fields of viewcapture a desired portion of the patient. For example, in someembodiments, face/torso acquisition module 264 may pan, tilt, or zoom avital sign acquisition camera 276 so that it is focused on a patient'sface and/or upper torso. Additionally or alternatively, face/torsoacquisition module 264 may pan, tilt, or zoom one vital sign acquisitioncamera 276 to capture predominantly the patient's face, and another topredominantly capture the patient's torso. Various vital signs and/orphysiological parameters may then be acquired. For instance, vital signssuch as the patient's pulse rate and SpO₂ may be obtained, e.g., byvital signs measurement module 266, by performing image processing on anvideo of the patient's face captured by vital sign acquisition camera(s)276. Vital signs and/or physiological parameters such as the patient'srespiratory rate, and so forth may be obtained, e.g., by vital signsmeasurement module 266, by performing image processing on an video ofthe patient's torso captured by vital sign acquisition camera(s) 276. Ofcourse, the face and torso are just two examples of body portions thatmay be examined to obtain vital signs, and are not meant to be limiting.

Deterioration detection module 268 may be configured to analyze varioussignals and/or data to determine whether a condition of a registeredpatient (or even non-registered companions) is deteriorating, improving,and/or remaining stable. In some embodiments, the patient condition maybe represented, at least in part, by the same patient acuity measuresdescribed above for determining order of patients for monitoring. Assuch, the deterioration detection module 268 may include one or moreCDS, case-based reasoning, or other clinical reasoning algorithms asdescribed herein or other clinical reasoning algorithms (e.g., trainedlogistic regression models or other machine learning models) forassessing patient condition measures other than acuity measuresdescribed herein. In some embodiments, the algorithms for assessingpatient acuity or other measures of patient condition employed by thedeterioration detection module 268 may be updated from time to time by,for example, writing new trained weights (e.g., theta values) for aselected machine learning module or providing new instructions forexecution by a processor (e.g. in the form of a java archive, JAR, fileor compiled library). These signals may include, for instance, apatient's initial vital signs and other physiological information (e.g.,obtained at blocks 108-110 of FIG. 1), updated vital signs obtained byvital signs measurement module 266, a patients initial patient acuitymeasure (e.g., calculated during registration), and/or a patient'supdated patient acuity measure (e.g., calculated based on updated vitalsigns and/or physiological parameters received from vital signsmeasurement module 266).

Based on determinations made using these data and/or signals,deterioration detection module 268 may send various alerts to variousother modules to take various actions. For example, deteriorationdetection module 268 may publish an alert, e.g., by sending the alert toEPS module 270 so that EPS module can publish the alert to subscribedmodules, such as alarm module 248 of hospital information system 240. Insome embodiments, such an alert may include, for instance, a patient'sname (or more generally, a patient identifier), a picture, live videostream, the patient's last detected location in the waiting room,baseline vital signs, one or more updated vital signs, and/or anindication of a patient acuity measure. On receipt of the alert, alarmmodule 248 may raise an alert or alarm to medical personnel of thepatient's deterioration and, among other things, the patient's lastdetected location in the waiting room.

EPS module 270 may be a general communication hub that is configured todistribute events released by various other components of FIG. 2. Insome embodiments, all or at least some of the other modules depicted inFIG. 2 may generate events that indicate some form ofresult/determination/computation/decision from that module. These eventsmay be sent, or “published,” to EPS module 270. All or some of the othermodules depicted in FIG. 2 may elect to receive, or “subscribe to,” anyevent from any other module. When EPS module 270 receives an event, itmay send data indicative of the event (e.g., forward the event) to allmodules that have subscribed to that event.

In some embodiments, EPS module 270 may be in communication with one ormore databases, such as database 272 and/or archive 274 (which may beoptional). In some embodiments, EPS module 270 may accept remoteprocedure calls (“RPC”) from any module to provide access to informationstored in one or more databases 272 and/or 274, and/or to addinformation (e.g., alerts) received from other modules to databases 272and/or 274. Database 272 (which may be the same as subject referencedatabase 412 in some embodiments) may store information contained inalerts, publications, or other communications sent/broadcast/transmittedby one or more other modules in FIG. 2. In some embodiments, database272 may store, for instance, subject reference templates associated withpatients and/or their initial vital signs, updated vital signs (acquiredby vital sign acquisition camera 276), and/or patient acuity measures.Optional archive 274 may in some embodiments store the same or similarinformation for a longer period of time.

It will be apparent that various hardware arrangements may be utilizedto implement the patient monitoring system 252. For example, in someembodiments, a single device may implement the entire system 252 (e.g.,a single server to operate the camera 276 to perform the vital signsacquisition functions 260-266 and to perform the vital sign(s) analysisand alerting functions including deterioration detection 268 and patientqueue management 258). In other embodiments, multiple independentdevices may form the system 252. For example, a first device may drivethe vital sign acquisition camera 276 and implement functions 260-266while another device(s) may perform the remaining functions. In somesuch embodiments, one device may be local to the waiting room whileanother may be remote (e.g., implemented as a virtual machine in ageographically distant cloud computing architecture). In someembodiments, a device (e.g., including a processor and memory) may bedisposed within the vital sign acquisition camera 276 itself and, assuch, the camera 276 may not simply be a dumb peripheral and, insteadmay perform the vital signs functions 260-266. In some such embodiments,another server may provide indications (e.g. identifiers, full records,or registered facial images) to the camera 276 to request that vitals bereturned for further processing. In some such embodiments, additionalfunctionality may be provided on-board the camera 276 such as, forexample, the deterioration detection 268 (or preprocessing therefor)and/or patient queue module 258 management may be performed on-board thecamera 276. In some embodiments, the camera 276 may even implement theHIS interface 250 or EPS 270. Various additional arrangements will beapparent.

FIG. 3 illustrates an example scenario in which disclosed techniques maybe implemented to identify patients 378A-C in a waiting room 304 formonitoring purposes. In this example, three patients 378A-C are waitingin a hospital waiting room 304 to be attended to by medical personnel380. Two video cameras 376A, 376B are mounted on a surface (e.g.,ceiling, wall) of waiting room 304. The two video cameras 376A, 376B maybe used to monitor patients 378 in waiting room 304. The patients 378A-Cmay each be assigned a patient acuity measure by triaging medicalpersonnel (not depicted) based on a preliminary patient conditionanalysis. As the patients 378 wait for an attending physician, the twovideo cameras 376A, 376B may capture digital image(s) that are analyzedusing techniques described herein to identify patients selected formonitoring. The same video cameras (assuming they are configured tounobtrusively acquire vital signs) or different video cameras may thenbe operated to monitor patients 378 as described above, e.g., to detectpatient deterioration. In some embodiments, a patient acuity measureassociated with a patient may be updated by medical personnel inresponse to detection by patient monitoring system (more specifically,deterioration detection module 268) that a patient has deteriorated. Invarious embodiments, when a new patient enters waiting room 304, a newround of patient monitoring and prioritization may be performed, e.g.,by patient monitoring system 252. The patient queue may be automaticallyupdated, e.g., by patient queue module 258, each time a new patiententers waiting room 304. Additionally or alternatively, medicalpersonnel may manually update the patient queue to include anewly-arrived patient after triaging.

Techniques described herein are not limited to hospital waiting rooms.There are numerous other scenarios in which techniques described hereinmay be implemented to identify/locate subjects in digital images orvideos. For example, disclosed techniques may also be used for securitymonitoring of crowds in airports, arenas, border crossings, and otherpublic places. In such scenarios, rather than monitoring patients todetermine patient acuity measures, subjects may be identified for otherpurposes, such as risk assessments or post-event investigation.Techniques described herein may also be applicable in scenarios such asin fitness environments (e.g., gyms, nursing homes) or othersurveillance scenarios (e.g., airports, border crossings, etc.) in whichidentification of individual subjects depicted in digital images may beimplemented. For example, in airports, subjects waiting at gates couldbe identified, for example, by comparing images of subjects waiting atgates to subject reference templates obtained at checkin. In addition,techniques described herein may be used to identify patients who leftwithout being seen, without requiring that patients' faces be visible.

FIG. 4 schematically depicts, at a relatively high level, an example ofcomponents configured with selected aspects of the present disclosure,as well as example interactions between those components. In variousembodiments, one or more of these components may be implemented usingany combination of hardware and software, e.g., as part of patientmonitoring system 252 in FIG. 2. For example, the components of FIG. 4may be used at block 108 of FIG. 1 to register a subject such as apatient in a subject reference database 412. Along with the subjects'intake information (e.g., age, gender, name, initial vital signs, etc.),any number of “subject reference templates” that comprise digital imagesof the subject's face from multiple views (e.g., different angles,different facial expressions, different lighting conditions, differenthead positions, etc.) may be selected and associated with the subject inthe subject reference database 412, e.g., by way of a medical recordnumber (“MRN”). These subject reference templates may then be usedlater, e.g., by patient identification module 260, to identify thesubject in an area such as a waiting room using another camera (e.g.,vital sign acquisition cameras 276, 376) that captures the waiting roomin its field of view. Once the subject is identified, the subject'slocation can be used for various purposes, such as being contacted bymedical personnel, having vital signs unobtrusively acquired, etc.

Starting at bottom right, an intake routine 402 is depicted thatincludes operations for intake of a newly-registered subject (e.g.,registering and/or triaging a new patient) and adding that subject to asubject reference database 412, in accordance with various embodiments.A first camera 456 may be configured to capture one or more of what willbe referred to herein as “intake” digital images 404 (e.g., individualimages and/or a stream of images such as a video stream). First camera456, which may correspond to camera 256 in FIG. 2 in some instances, maytake various forms, such as a webcam positioned in the intake area(e.g., registration and/or triage), a camera integral with a computingdevice operated by intake personnel (e.g., a triage nurse), etc. Thisimage capture may be un-intrusive to both the intake personnel and thesubject, as it may occur automatically with little or no humanintervention (although this is not meant to be limiting).

At block 406, intake digital image(s) 404 may be analyzed, e.g., by oneor more computing systems operably coupled with camera 456 (e.g.,patient capture module 254 in FIG. 2) to detect one or more portions ofdigital images 404 that depict a face of a subject currently located inan intake area (e.g., registration and/or triage). FIG. 6 demonstratesone example technique for detecting the subject's face.

At block 408, a subset of intake digital images that depict multipledifferent views of a face of the subject may be selected from pluralityof intake digital images 404. The selected subset may be used togenerate subject reference templates that are used to visuallyidentify/locate the subject later. In some embodiments, the subset ofintake digital images used to generate the subject reference templatesmay be selected based on being sufficiently dissimilar to one or moreother intake digital images. FIGS. 5 and 8 below demonstrate exampletechniques for selecting subsets of intake images for generation ofsubject reference templates.

At block 410, the generated subject reference templates may be stored,e.g., in subject reference database 412, in association with thesubject. In various embodiments, the generated subject referencetemplates may be stored in subject reference database 412 in associationwith information related to the subject, e.g., by way of theaforementioned MRN. More generally, subject reference database 412 maystore subject reference templates related to a plurality of subjects,such as a plurality of registered patients in waiting room 104 that maybe awaiting medical treatment.

Moving to top right, a subject (e.g., patient) monitoring routine 414 isdepicted that demonstrates one example of how a particular subject(e.g., patient) may be selected, e.g., by medical personnel such as aduty nurse and/or automatically (e.g., based on a patient acuity measureof the subject), as well as how a query may be issued that seeks tolocate the subject in an area such as a waiting room, in accordance withvarious embodiments. The subject under consideration will heretofore bereferred to as the “queried subject.”

At block 416, subject reference templates associated with the queriedsubject may be retrieved from subject reference database 412, e.g., bypatient identification module 260. Meanwhile, as part of an ongoingpatient identification routine 418 that may be performed, for instance,by patient identification module 260 of FIG. 2, another camera 476,which may or may not take the form of a vital sign acquisition cameradescribed previously, may be acquiring digital images 420 that depict anarea in which the queried patient is believed to be, such as waitingroom 104.

At block 422, one or more portions of the digital image(s) 420 thatdepict faces of one or more subjects in the area may be detected, e.g.,by patient identification module 260, as what will be referred to hereinas “detected face images.” In various embodiments, the operations ofblock 422 may be performed continuously and/or may be triggered byreceipt of the subject query from patient monitoring routine 414.Similar techniques for face detection may be applied at block 422 aswere applied at block 406, and will be described in more detail below.

At block 424, one or more operations may be performed to normalize thefaces depicted in the portions detected at block 422. For example, insome embodiments, geometric warping and/or other similar techniques maybe employed to normalize detected faces to be at or near frontal views.FIG. 7 below demonstrates one example technique for normalizing detectedfaces. Thus, the output of block 424 may be a series of normalizeddetected face images.

In some embodiments, at block 426, a “first pass” of the normalizeddetected faces may be performed to obtain a preliminary match of thequeried subject. For example, in some implementations, each of thedetected face images may be applied as input across a trained machinelearning model. In various embodiments, the machine learning model maytake various forms, such as a linear discriminant analysis model, asupport vector machine, a neural network, and so forth. In variousembodiments, the machine learning model may be trained and/ordynamically retrained, e.g., at block 419, using the subject referencetemplates currently stored in subject reference database 412. In variousembodiments, output generated via the machine learning model may includesimilarity scores between each input detected face image and eachsubject, or may include the subject that is most similar to the detectedface image. For a given normalized detected face image, the registeredsubject that yields the highest similarity score (e.g., that satisfiessome preset minimum threshold) may be identified as a match. Forexample, at block 428 of FIG. 4, it may be determined whether a queriedsubject is identified in a given detected face image. If the answer isno (e.g., a minimum threshold is not satisfied), then control may passback to block 424 and the next detected face image may be normalized andapplied as input across the machine learning model.

The machine learning model may be trained and/or dynamically retrainedat block 419 at various times. In some embodiments, whenever a newpatient record is created in subject reference database 412 or anexisting patient is released from subject reference database 412,subject reference database 412 may publish an event, e.g., to EPS module270. In response, EPS module 270 may trigger training of a new machinelearning model, or retrain an existing machine learning model, based onthe subject reference templates currently stored in subject referencedatabase 412. In some contexts, such as in hospitals, this may befeasible because the number of patients in one day is generally not verylarge. Thus, a multi-class linear discriminant analysis machine learningmodel may be used because it is relatively inexpensive computationallyto retrain, and thus can be retrained in near real time.

Back at block 428, if the answer is yes, then in some embodiments, a“second pass” of testing may be applied to the normalized detected faceimage. For example, at block 430, so-called “pose-adaptive face imagematching” may be applied in which the normalized detected face image iscompared to each of the subject reference templates associated with thequeried subject. FIG. 9 below demonstrates one example technique forperforming pose-adaptive face image matching. As will be describedbelow, pose-adaptive face image matching may be an iterative processthat repeats a predetermined number of times to determine whether thenormalized detected face image matched at blocks 426-428 is truly thatof the queried subject. In other words, the operations of block 430 mayserve to corroborate the initial finding of blocks 426-428.

At block 432, if it is determined that the normalized detected faceimage truly depicts the queried subject, then at block 436, the locationassociated with the normalized detected face image (e.g., a particularlocation such as a seat in a waiting room at which the subject islocated) may be provided as output. On the other hand, if corroborationis not possible, e.g., because some predetermined similarity thresholdis not met during the iterative pose-adaptive face image matchingprocess, then at block 434, camera 476 may be repositioned (e.g.,panned, tilted, zoomed) to focus on a different area, e.g., thatcontains different subject(s).

FIG. 5 depicts one example of how various aspects of the workflow ofintake routine 402 may be implemented, in accordance with variousembodiments. As described above, camera 456 may acquire intake digitalimages 404, e.g., as a video stream. In some embodiments, intake digitalimages 404 may depict an intake (e.g., triage) area, although this isnot required. The operations depicted in FIG. 5 may be performed atvarious computing devices, such as a computing device that is operablycoupled with camera 456 in or near the intake area.

In the intake (e.g., triage) area where a new subject is assessed (e.g.,clinically assessed), for each new intake digital image (e.g., frame ofa video stream) captured by camera 456, at blocks 502 and 504,respectively, face detection (e.g., of a new face) and face tracking(e.g., of a face detected in a previous intake digital image) may beperformed in parallel. This ensures that a face of each subject in theintake area is detected, no matter which subject entered first. For eachnewly detected face, at block 506, a new face tracker is launched. Thisnew face tracker will start its analysis at the next image frame. Then,at block 508, the newly detected face is normalized, e.g., to anear-frontal view (normalization is demonstrated in more detail in FIG.7).

In some embodiments, this normalized detected face may be deemed asubject template candidate. Then, the new subject reference templatecandidate may be compared, e.g., at block 510, with existing subjectreference template candidates (e.g., acquired from previous imageframes), if any yet exist. Various criteria may be used to determinewhether to keep the new subject reference template candidate, e.g., as areplacement of another previously-captured subject reference templatecandidate, or to discard the new subject reference template candidate.Ultimately, only the most representative subject reference templatescandidates may be selected and retained in subject reference database412. FIG. 8 demonstrates, in greater detail, one example of how intakedigital images may be selected (510) for use in generating subjectreference templates.

Turning now to face tracking block 504, for each tracked face previouslydetected in each intake image frame, at block 512, it may be determinedwhether the corresponding subject is leaving the camera's field of view.FIG. 6 depicts one example of how a determination may be made of whethera subject is leaving. If the answer at block 512 is yes, then operationpasses back to block 504 and the next tracked face is selected. If theanswer at block 512 is no, then at block 514, homography estimation maybe performed, e.g., to estimate a three-dimensional head pose of thetracked face in the current intake image frame. Based on the estimatedpose, the tracked face image in the current frame may be “frontalized”(removing the pose effect on face appearance) at block 516. Control maythen pass to block 508.

FIG. 6 demonstrates one example technique for detecting a subject'sface, e.g., during intake (e.g., at block 406) or later during subjectmonitoring (e.g., at block 422). A camera's field of view (“FOV”) 640 isshown, and may be associated with any camera described herein, such ascamera 456 or camera 476. FIG. 6 illustrates the both detection of asubject (642A) entering and a subject (642B) leaving. Both situationsonly happen when the subject's face is partially visible in FOV 640. Thepresence of a subject may be detected, for instance, by measuring theoverlapping ratio of a face region to FOV 640. If the ratio is less thana particular number, such as one, and is increasing compared to theprevious frame(s), the subject may be determined to be entering.Otherwise, if the ratio is greater than one and is decreasing comparedto the previous frame(s), the subject may be determined to be leaving.If either of the two situations lasts for a predetermined time interval,such as five seconds, it is possible to determine that the subject hasentered or left.

FIG. 7 depicts details of one example face normalization routine, e.g.,that may be performed at block 424 of FIG. 4 and/or block 508 of FIG. 5.Input may take the form of a detected face image, e.g., from block 422of FIG. 4 and/or from block 506/516 of FIG. 5. Output may be anormalized detected face image. At blocks 702 and 704, left and righteye detection operations may be performed (operations 702 and 704 mayalso be performed in the reverse order, or in parallel). Theseoperations may include a variety of image processing techniques, such asedge detection, template matching, Eigenspace methods, Hough transforms,morphological operations, trained neural networks, etc. At block 706, ifboth eyes are successfully detected, control may pass to block 714, atwhich point the face may be normalized (e.g., geometric warping may beapplied to the detected face image to make the face approximatelyfrontal facing). From block 714, control may pass, for instance, toblock 426 of FIG. 4 or to block 510 of FIG. 5.

If the answer at block 706 is no, then at block 708 it may be determinedwhether either eye was detected. If the answer is no, then control maypass downstream of operation 714, in some instances a failure event maybe raised, and then control may proceed, e.g., to block 426 of FIG. 4 orto block 510 of FIG. 5. If only one eye was successfully detected atblocks 702-704, then at block 710, the detected eye region may bemirrored horizontally, and the mirror eye patch may be searched, e.g.,using template matching, to locate the other eye. Then, operation mayproceed to block 714, which was described previously.

FIG. 8 depicts one example of how detected face images may be selectedas subject reference templates, e.g., for inclusion in subject referencedatabase 412, at block 408 of FIG. 4 and block 510 of FIG. 5. Controlmay pass to the operations of FIG. 8 from various locations, such asblock 406 of FIG. 4, block 508 of FIG. 5 (if the detected face imageunder consideration is newly detected in the current intake digitalimage frame), and/or block 516 of FIG. 5 (if the detected face imageunder consideration was detected in a prior intake digital image frameand is currently being tracked). At block 802, it may be determinedwhether the face is occluded. If the answer is yes, then control maypass to block 504, at which point the next tracked face (if any) may beanalyzed.

If the answer at block 802 is no, then at block 806, image similaritiesbetween the current detected face image and any existing subjectreference templates for the current subject may be determined. At block808, it may be determined whether there are yet enough subject referencetemplates collected for the current subject. Various numbers of subjectreference templates may be selected for each new subject. In someembodiments, as many as nine subject reference templates may becollected. While collecting more subject reference templates isfeasible, diminishing returns may be experienced after some point.

If there are not yet enough subject reference templates collected forthe current subject, then at block 408/410 (same as FIG. 4), the currentdetected face image may be used to generate a subject reference templatethat is then added to subject reference database 412. However, at block808, if there are already enough templates collected, then in someembodiments, it may be determined whether the current detected faceimage is sufficiently different from previously-collected subjectreference templates of the current subject to warrant replacing apreviously-collected subject reference template. For example, at block,at block 812, a determination may be made of whether the currentdetected face image is more dissimilar from each previously-collectedsubject reference template than any of the previously-collected subjectreference templates are from each other. If the answer is yes for aparticular subject reference template, then the current detected faceimage may be used to generate a new subject reference template thatreplaces the particular subject reference template in subject referencedatabase 412.

The operations of FIG. 8 (and more generally, the operations of FIG. 5)are repeated for every intake digital image captured by camera 456, andeach subject may be tracked, for instance, until they leave the intakearea (block 512). Consequently, of the total number of intake digitalimages acquired while the subject is in FOV 640 of camera 456, the nintake digital images having the most suitably (e.g., most diverse)views may be selected to generate subject reference templates for thatparticular subject. As mentioned previously, these subject referencetemplates may be used later, e.g., in response to a subject beingqueried at subject monitoring routine 414.

FIGS. 5 and 8 relate to collecting subject reference templates for eachsubject to be stored in subject reference database 412. FIGS. 6 and 7relate to both to collecting subject reference templates and using thosesubject reference templates to identify subjects in areas downstreamfrom intake areas, such as hospital waiting rooms. FIG. 9 relates to thelatter. In particular, FIG. 9 depicts one example of operations that maybe performed as part of the pose-adaptive face image matching operationof block 430 in FIG. 4. As noted above, in some embodiments,pose-adaptive face image matching may constitute a “second pass” afterthe initial subject matching performed at block 426, e.g., using atrained machine learning model. In various embodiments, thepose-adaptive face image matching may provide more accurateidentification of a subject in an area (e.g., a patient in a waitingroom) than the operations of block 426 alone.

The process of pose-adaptive face image matching generally relates tomatching a detected face image (which may or may not be normalized) withone or more subject reference templates retrieved in response to asubject query, In particular, to eliminate or reduce matching errorcaused by spatial misalignment, detected face images may be repeatedlymatched to reference subject templates by increasing a matchingthreshold and iteratively aligning the detected face images with thesubject reference templates.

In FIG. 9, two inputs are received: the current detected face imageunder consideration and the subject reference templates associated withthe queried subject (i.e., the subject being searched for). At block902, both inputs may be used to perform a one-to-many matching betweenthe single detected face image and the multiple subject referencetemplates associated with the queried subject. In some embodiments, arespective similarity score may be calculated between the detected faceimage and each subject reference template. At block 904, it may bedetermined whether one or more of the similarity scores satisfies arelatively small similarity threshold. If the answer at block 904 is no,then it may be determined that the subject depicted in the currentdetected face image does not match the subject reference templates, andthe detected face image associated with the next detected subject in thearea being monitored may be selected.

If the answer at block 904 is yes, then at block 906, the subjectreference template that is most similar to the detected face image—e.g.,the subject reference template for which the highest similarity scorewas calculated at block 902—may be selected. At block 908, the selectedmost similar subject reference template and the single detected faceimage may be aligned. For example, in some embodiments, a geometricerror between the two may be calculated. Based on this geometric error,at block 910 the detected face image may be geometrically warped to thesubject reference template. In some embodiments, this process may beiterated until, at block 912, some similarity threshold is satisfied(e.g., 0.9), or, at block 914, some maximum number of iterations hasbeen reached. If the similarity threshold of block 912 is satisfied, amatch is found between the detected face image and the subjectassociated with the subject reference template, and the subject depictedin the detected face image is identified as the queried subject. But, ifthe maximum number of iterations is reached at block 914 withoutsatisfying this similarity threshold, then the subject depicted in thedetected face image is indicated to not match the subject associatedwith the subject reference template.

When the detected face image under consideration is not matched to thesubject reference template, in some embodiments, the camera (e.g., 276,376, 476) that is monitoring an area such as a waiting room may bepanned, tilted, and/or zoomed to capture another location of aparticular number of locations in which subjects such as patients arelikely to be found. These locations may correspond to, for instance,seats in a waiting room, exercise equipment in a gym, seats at anairport gate, etc. In some embodiments, PTZ control of camera 476 may bepre-calibrated to sequentially capture these locations. If the queriedsubject is not found at the current location, camera 476 may besequentially repositioned through the remainder of the present locationsuntil the queried subject has been found, or until all preset locationshave been scanned. Alternatively, if no preset locations are available,in some embodiments, people detection techniques may be employed todetermine locations of people generally in an area, and then each ofthose locations may be scanned. Additionally, if the queried subject isnot found, especially when the queried subject is a patient admitted toa hospital emergency department, then one or more notifications may besent to various personnel, such as hospital staff. In instances in whichthe absent subject is a patient being identified for unobtrusivemonitoring using one or more vital sign acquisition cameras 276, theabsent patient may be referred back to patient queue module 258 forreinsertion into the patient queue.

On the other hand, if the detected face image matches the subjectreference template(s), an output may be provided, e.g., by patientidentification module 260 to another module in FIG. 2 and/or to thepersonnel associated with subject monitoring 414 in FIG. 4. This outputmay indicate the location of the queried patient, e.g., in the waitingroom or other area being monitored. Although not depicted in FIG. 9, invarious embodiments, more than one detected face image may be acted uponat once, e.g., in parallel. For example, two or more separate detectedface images captured within a FOV of a camera may be processed inparallel.

FIG. 10 depicts an example method 1000 for practicing selected aspectsof the present disclosure, in accordance with various embodiments. Forconvenience, the operations of the flow chart are described withreference to a system that performs the operations. This system mayinclude various components of various computer systems, includingpatient monitoring system 252. Moreover, while operations of method 1000are shown in a particular order, this is not meant to be limiting. Oneor more operations may be reordered, omitted or added.

At block 1002, the system may acquire a plurality of intake digitalimages that capture at least a first subject. For example, in someembodiments, patient capture module 254 may acquire a plurality ofintake (e.g., pre-waiting room areas 102) digital images from camera256. Camera 256 may be located in an intake area such as a hospitalregistration/triage, a check-in desk at an airport or train station, acheck-in desk at a gym, an intake area associated with a bordercrossing, etc. In some embodiments, the plurality of intake digitalimages may include video frames that are captured for the entire timethe subject is in the intake area (e.g., from the moment they aredetected entering to the moment they are detected leaving), or for someother time interval, such as while triage is being performed, amanually-selected time interval, etc.

At block 1004, the system may select, from the plurality of intakedigital images, a subset of intake digital images that depict multipledifferent views of a face of the first subject. FIGS. 5 and 8 depictnon-limiting examples of how subject reference templates may be selectedfrom a plurality of intake images. Generally, the subject referencetemplates associated with a particular subject may be selected toprovide a variety of different views of the subject's face, such asdifferent facial expressions, different lighting conditions, differentposes, etc.

At block 1006, the system may generate, based on the selected subset ofintake digital images, first subject reference templates, and store themin subject reference database 412. In some embodiments, the subjectreference templates are the same digital images as the selected intakeimages. In other embodiments, however, the subject reference templatesmay be altered versions of the corresponding selected intake digitalimages, e.g., cropped, enhanced, etc. For example, each subjectreference template may include a sub-portion of (e.g., cropped from) acorresponding selected intake digital image, e.g., a sub-portiondepicting the subject's face. In various embodiments, the generatedfirst subject reference templates may be stored in subject referencedatabase 412 in association with information related to the firstsubject. More generally, subject reference database 412 may storesubject reference templates related to a plurality of subjects, such asall patients who have been registered and/or triaged on a given day, orduring a particular time interval.

At block 1008, the system may select a second subject to identify withinan area. For example, in the patient monitoring context, the patienthaving a patient acuity score that places them at the head of thepatient queue may be selected, e.g., at block 108 of FIG. 1.Additionally or alternatively, medical personnel may manually select apatient to identify, e.g., using a graphical user interface that depictsinformation about registered patients such as their acuity measures,pictures, etc. In FIG. 4 and elsewhere herein, the selected subject isoften referred to as the “queried subject.” In yet other embodiments,rather than attempting to locate a queried patient, the system maysimply scan through a sequence of preselected locations, e.g.,corresponding to a series of waiting room chairs, and using disclosedtechniques, attempt, at each location, to identify whichever subject isin that location (e.g., determine the MRN of the depicted subject).

At block 1010, the system may retrieve second subject referencetemplates related to the second subject from subject reference database412. For example, an MRN associated with the subject selected at block1008 may be provided as input to subject reference database 412. Thepreviously-collected subject reference templates associated with thatMRN may be provided as output.

At block 1012, one or more digital images that depict the area (e.g.,waiting room 104) may be acquired, e.g., by cameras 276, 346, 476, orother cameras. In the patient monitoring context, the camera(s) thatacquire the digital images at block 1012 may or may not be vital signacquisition cameras. In other contexts, the camera(s) that acquire thedigital images at block 1012 may be other types of cameras, and may ormay not have PTZ capabilities. As noted above, in various embodiments,there may be any number of cameras that acquire the digital imagescapturing the area. In some embodiments, the cameras may be constantlyand/or continuously capturing digital images of the area, and only thosedigital images that are captured, e.g., after the subject is selected atblock 1008, may be used for operations below.

At block 1014, the system may detect (or localize), as one or moredetected face images, one or more portions of the one or more digitalimages acquired at block 1012 that depict faces of one or more subjectsin the area. Various techniques may be used to perform face detection,including but not limited to deep learning, genetic algorithms and/orthe eigenface technique. For example, possible human eye regions may bedetected by testing all the valley regions in a gray-level digitalimage. Then, a genetic algorithm may be employed to generate all thepossible face regions which include, for instance, the eyebrows, theiris, the nostril and/or the mouth corners. Additionally oralternatively, various other techniques may be employed, such astemplate matching, scale-invariant feature transform (“SIFT”), lineardiscriminant analysis, elastic bunch graph matching, hidden Markovmodels, etc. As noted above, in some embodiments, a number ofpredetermined locations within the area being monitored (e.g., waitingroom 104) may be established/selected as locations likely to containsubjects. In the hospital waiting room context or the airport gatecontext, the predetermined locations may correspond to seats in thearea. In various embodiments, one or more cameras (e.g., 276, 376, 476)may perform PTZ operations to sequentially scan each location, e.g.,performing one or more of the above-described operations, to detectdepicted faces.

At block 1016, a given detected face image of the detected one or moredetected face images may be compared to the second subject referencetemplates. For example, a detected face image from a first location of aplurality of predetermined locations may be selected. Additionally oralternatively, a last-known location of the currently queried subjectmay be selected first. Examples of the types of comparison that may beperformed include the “first past” machine learning model approachdescribed above in relation to block 426 of FIG. 4, and/or the “secondpass” approach described above in relation to block 430, which in someembodiments may constitute pose-adaptive face image matching asdemonstrated in FIG. 9.

At block 1018, the system may identify, based on the comparing, thesecond subject in the one or more digital images that capture the area.For example, if the given detected face image (which as described abovemay be associated with a location such as a seat in waiting room 104) isdetermined to depict the queried subject, then the location associatedwith the given detected face image may be provided as the location ofthe queried patient. This location may be used for various purposes. Forexample, one or more vital sign acquisition cameras (which may or maynot be the same cameras the captured the digital images waiting room104) may acquire one or more vital signs from the subject at thelocation, as described above. Additionally or alternatively, if thesubject was queried manually, e.g., by medical or other personnel, thenthe queried subject's location may be provided as output. In someembodiments, the output location may be provided textually, e.g.,“<subject> is sitting in seat 13.” In other embodiments, the outputlocation may be used within a graphical user interface (e.g., operatedby a duty nurse or other personnel) to annotate a visual rendition ofthe area being monitored. For example, the queried subject may bevisually emphasized, e.g., with a bounding box, or otherwise renderedmore conspicuously than other subjects in the area.

FIG. 11 is a block diagram of an example computer system 1110. Computersystem 1110 typically includes at least one processor 1114 whichcommunicates with a number of peripheral devices via bus subsystem 1112.As used herein, the term “processor” will be understood to encompassvarious devices capable of performing the various functionalitiesattributed to components described herein such as, for example,microprocessors, GPUs, FPGAs, ASICs, other similar devices, andcombinations thereof. These peripheral devices may include a dataretention subsystem 1124, including, for example, a memory subsystem1125 and a file storage subsystem 1126, user interface output devices1120, user interface input devices 1122, and a network interfacesubsystem 1116. The input and output devices allow user interaction withcomputer system 1110. Network interface subsystem 1116 provides aninterface to outside networks and is coupled to corresponding interfacedevices in other computer systems.

User interface input devices 1122 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touchscreen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and/or othertypes of input devices. In general, use of the term “input device” isintended to include all possible types of devices and ways to inputinformation into computer system 1110 or onto a communication network.

User interface output devices 1120 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem may also provide non-visual display such as via audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computer system 1110 to the user or to another machine or computersystem.

Data retention system 1124 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein. For example, the data retention system 1124 may include thelogic to perform selected aspects of FIGS. 4-10, and/or to implement oneor more components of patient monitoring system 252, including patientidentification module 260, patient capture module 254, etc.

These software modules are generally executed by processor 1114 alone orin combination with other processors. Memory 1125 used in the storagesubsystem can include a number of memories including a main randomaccess memory (RAM) 1130 for storage of instructions and data duringprogram execution, a read only memory (ROM) 1132 in which fixedinstructions are stored, and other types of memories such asinstruction/data caches (which may additionally or alternatively beintegral with at least one processor 1114). A file storage subsystem1126 can provide persistent storage for program and data files, and mayinclude a hard disk drive, a floppy disk drive along with associatedremovable media, a CD-ROM drive, an optical drive, or removable mediacartridges. The modules implementing the functionality of certainimplementations may be stored by file storage subsystem 1126 in the dataretention system 1124, or in other machines accessible by theprocessor(s) 1114. As used herein, the term “non-transitorycomputer-readable medium” will be understood to encompass both volatilememory (e.g. DRAM and SRAM) and non-volatile memory (e.g. flash memory,magnetic storage, and optical storage) but to exclude transitorysignals.

Bus subsystem 1112 provides a mechanism for letting the variouscomponents and subsystems of computer system 1110 communicate with eachother as intended. Although bus subsystem 1112 is shown schematically asa single bus, alternative implementations of the bus subsystem may usemultiple busses. In some embodiments, particularly where computer system1110 comprises multiple individual computing devices connected via oneor more networks, one or more busses could be added and/or replaced withwired or wireless networking connections.

Computer system 1110 can be of varying types including a workstation,server, computing cluster, blade server, server farm, or any other dataprocessing system or computing device. In some embodiments, computersystem 1110 may be implemented within a cloud computing environment. Dueto the ever-changing nature of computers and networks, the descriptionof computer system 1110 depicted in FIG. 11 is intended only as aspecific example for purposes of illustrating some implementations. Manyother configurations of computer system 1110 are possible having more orfewer components than the computer system depicted in FIG. 11.

While several embodiments have been described and illustrated herein,those of ordinary skill in the art will readily envision a variety ofother means and/or structures for performing the function and/orobtaining the results and/or one or more of the advantages describedherein, and each of such variations and/or modifications is deemed to bewithin the scope of the embodiments described herein. More generally,those skilled in the art will readily appreciate that all parameters,dimensions, materials, and configurations described herein are meant tobe exemplary and that the actual parameters, dimensions, materials,and/or configurations will depend upon the specific application orapplications for which the teachings is/are used. Those skilled in theart will recognize, or be able to ascertain using no more than routineexperimentation, many equivalents to the specific embodiments describedherein. It is, therefore, to be understood that the foregoingembodiments are presented by way of example only and that, within thescope of the appended claims and equivalents thereto, embodiments may bepracticed otherwise than as specifically described and claimed.Inventive embodiments of the present disclosure are directed to eachindividual feature, system, article, material, kit, and/or methoddescribed herein. In addition, any combination of two or more suchfeatures, systems, articles, materials, kits, and/or methods, if suchfeatures, systems, articles, materials, kits, and/or methods are notmutually inconsistent, is included within the scope of the presentdisclosure.

All definitions, as defined and used herein, should be understood tocontrol over dictionary definitions, definitions in documentsincorporated by reference, and/or ordinary meanings of the definedterms.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, i.e., elements that are conjunctively present in some casesand disjunctively present in other cases. Multiple elements listed with“and/or” should be construed in the same fashion, i.e., “one or more” ofthe elements so conjoined. Other elements may optionally be presentother than the elements specifically identified by the “and/or” clause,whether related or unrelated to those elements specifically identified.Thus, as a non-limiting example, a reference to “A and/or B”, when usedin conjunction with open-ended language such as “comprising” can refer,in one embodiment, to A only (optionally including elements other thanB); in another embodiment, to B only (optionally including elementsother than A); in yet another embodiment, to both A and B (optionallyincluding other elements); etc.

As used herein in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of” or “exactly one of,” or, when usedin the claims, “consisting of,” will refer to the inclusion of exactlyone element of a number or list of elements. In general, the term “or”as used herein shall only be interpreted as indicating exclusivealternatives (i.e. “one or the other but not both”) when preceded byterms of exclusivity, such as “either,” “one of,” “only one of,” or“exactly one of.” “Consisting essentially of,” when used in the claims,shall have its ordinary meaning as used in the field of patent law.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in one embodiment, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in another embodiment, to at least one, optionally includingmore than one, B, with no A present (and optionally including elementsother than A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

It should also be understood that, unless clearly indicated to thecontrary, in any methods claimed herein that include more than one stepor act, the order of the steps or acts of the method is not necessarilylimited to the order in which the steps or acts of the method arerecited.

In the claims, as well as in the specification above, all transitionalphrases such as “comprising,” “including,” “carrying,” “having,”“containing,” “involving,” “holding,” “composed of,” and the like are tobe understood to be open-ended, i.e., to mean including but not limitedto. Only the transitional phrases “consisting of” and “consistingessentially of” shall be closed or semi-closed transitional phrases,respectively, as set forth in the United States Patent Office Manual ofPatent Examining Procedures, Section 2111.03. It should be understoodthat certain expressions and reference signs used in the claims pursuantto Rule 6.2(b) of the Patent Cooperation Treaty (“PCT”) do not limit thescope.

1. A method implemented by one or more processors, the methodcomprising: acquiring a plurality of intake digital images that captureat least a first subject; selecting, from the plurality of intakedigital images, a subset of intake digital images that depict multipledifferent views of a face of the first subject; generating, based on theselected subset of intake digital images, first subject referencetemplates, wherein the first subject reference templates are stored in asubject database in association with information related to the firstsubject, and the subject database stores subject reference templatesrelated to a plurality of subjects; selecting a second subject toidentify within an area; retrieving second subject reference templatesrelated to the second subject from the subject reference database;acquiring one or more digital images that depict the area; detecting, asone or more detected face images, one or more portions of the one ormore digital images that depict faces of one or more subjects in thearea; comparing a given detected face image of the detected one or moredetected face images to the second subject reference templates; andidentifying, based on the comparing, the second subject in the one ormore digital images that capture the area.
 2. The method of claim 1,wherein the area comprises a waiting room, the intake images areacquired using a first camera that is configured to capture aregistration or triage area, and the digital images that depict thewaiting room are acquired using a second camera that is configured tocapture the waiting room.
 3. The method of claim 1, wherein thecomparing comprises applying the given detected face image as inputacross a trained machine learning model to generate output thatindicates a measure of similarity between the given detected face imageand the second subject, wherein the machine learning model is trainedbased at least in part on the second subject reference templates.
 4. Themethod of claim 3, wherein the trained machine learning model comprisesa linear discriminant analysis model.
 5. The method of claim 4, furthercomprising retraining the machine learning model in response to a newsubject being added to the subject database or an existing subject beingremoved from the subject database.
 6. The method of claim 3, wherein thetrained machine learning model is trained based on the subject referencetemplates related to the plurality of subjects.
 7. The method of claim1, wherein one or more of the subset of intake digital images areselected based on being sufficiently dissimilar to one or more otherintake digital images.
 8. The method of claim 1, further comprisingnormalizing the one or more face images so that each detected face imagedepicts a frontal view of a face.
 9. The method of claim 8, wherein thenormalizing includes geometric warping.
 10. A system comprising one ormore processors and memory operably coupled with the one or moreprocessors, wherein the memory stores instructions that, in response toexecution of the instructions by one or more processors, cause the oneor more processors to: acquire a plurality of intake digital images thatcapture at least a first subject; select, from the plurality of intakedigital images, a subset of intake digital images that depict multipledifferent views of a face of the first subject; generate, from theselected subset of intake digital images, first subject referencetemplates, wherein the first subject reference templates are stored in asubject database in association with information related to the firstsubject, and the subject database stores subject reference templatesrelated to a plurality of subjects; select a second subject to identifywithin an area; retrieve second subject reference templates related tothe second subject from the subject reference database; acquire one ormore digital images that depict the area; detect, as one or moredetected face images, one or more portions of the one or more digitalimages that depict faces of one or more subjects in the area; compare agiven detected face image of the detected one or more detected faceimages to the second subject reference templates; and identify, based onthe comparing, the second subject in the one or more digital images thatcapture the area.
 11. The system of claim 10, wherein the area comprisesa waiting room, the intake images are acquired using a first camera thatis configured to capture a registration or triage area, and the digitalimages that depict the waiting room are acquired using a second camerathat is configured to capture the waiting room.
 12. The system of claim10, further comprising instructions to apply the given detected faceimage as input across a trained machine learning model to generateoutput that indicates a measure of similarity between the given detectedface image and the second subject, wherein the machine learning model istrained based at least in part on the second subject referencetemplates.
 13. The system of claim 12, wherein the trained machinelearning model comprises a linear discriminant analysis model. 14.(canceled)
 15. (canceled)
 16. (canceled)
 17. (canceled)
 18. (canceled)19. At least one non-transitory computer-readable medium comprisinginstructions that, in response to execution of the instructions by oneor more processors, cause the one or more processors to perform thefollowing operations: acquiring a plurality of intake digital imagesthat capture at least a first subject; selecting, from the plurality ofintake digital images, a subset of intake digital images that depictmultiple different views of a face of the first subject; generating,based on the selected subset of intake digital images, first subjectreference templates, wherein the first subject reference templates arestored in a subject database in association with information related tothe first subject, and the subject database stores subject referencetemplates related to a plurality of subjects; selecting a second subjectto identify within an area; retrieving second subject referencetemplates related to the second subject from the subject referencedatabase; acquiring one or more digital images that depict the area;detecting, as one or more detected face images, one or more portions ofthe one or more digital images that depict faces of one or more subjectsin the area; comparing a given detected face image of the detected oneor more detected face images to the second subject reference templates;and identifying, based on the comparing, the second subject in the oneor more digital images that capture the area.
 20. The at least onenon-transitory computer-readable medium of claim 19, wherein the areacomprises a waiting room, the intake images are acquired using a firstcamera that is configured to capture a registration or triage area, andthe digital images that depict the waiting room are acquired using asecond camera that is configured to capture the waiting room.